Skip to content

chore(nix): unify Linux cross-compilation toolchains - #3242

Merged
SDAChess merged 5 commits into
mainfrom
feat/nix-cross-compiling-and-sysroot
Sep 10, 2026
Merged

SDAChess merged 5 commits into
mainfrom
feat/nix-cross-compiling-and-sysroot

Conversation

@SDAChess

@SDAChess SDAChess commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Replace the separate GNU and musl Nix development shells with one default shell
providing explicit Linux target toolchains and a native Darwin toolchain on macOS.
Release binary builds select their toolchain with Cargo's --target and use the
resulting binary directly, without post-link rewriting.

Related Issue

No issue linked yet. Issue linkage remains to be resolved before marking this
draft ready.

Changes

  • Provide x86_64 and aarch64 Linux GNU/musl toolchains through mkToolchain, with
    shared environment wiring and separate Linux and Darwin compiler wrappers.
  • Group the glibc 2.28 build, GCC environment, and sysroot assembly together.
    Build static Z3 and AWS-LC libraries with each target's stdenv.
  • Use the unprocessed Apple SDK for native Darwin linking and prevent the Rust
    toolchain from propagating replacement system libraries into the shell.
  • Remove the CI dev-shell input and matrix fields. Build release binaries with
    cargo auditable build --target <triple> and verify/upload artifacts from
    target/<triple>/release/.
  • Remove Linux and Darwin post-link rewriting, platform-specific linkage checks,
    and the unused interpreter input. Retain binary --version and Syft metadata
    checks. Branch checks continue to use Cargo's default host target.
  • Update the build architecture documentation.

Testing

  • Evaluated default shells for x86_64 Linux, aarch64 Linux, and aarch64 Darwin;
    all three derivations remain identical across the toolchain refactoring.
  • nix fmt and git diff --check pass.
  • Actionlint passes for the updated build/E2E/conformance workflows.
    Release Dev/Tag input removal was checked with ShellCheck disabled because
    those workflows have existing unrelated ShellCheck findings.
  • Run CI against the final commits, including native macOS builds and host
    builds/tests in branch-checks.
  • Full unit/E2E validation of the final changes.

Checklist

  • Follows Conventional Commits.
  • Commits include DCO signoff.
  • Architecture/build documentation updated.
  • Binary build workflows use the default shell and explicit Cargo targets.
  • Final commits pushed and CI results reviewed.

@copy-pr-bot

copy-pr-bot Bot commented Sep 9, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@SDAChess

SDAChess commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/ok-to-test 7e53559

@SDAChess

SDAChess commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/ok-to-test 675411f

@SDAChess

SDAChess commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/ok-to-test dc6a328

@SDAChess

SDAChess commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/ok-to-test 6067054

Replace the separate GNU and musl development shells with one shell that
provides explicit toolchains for x86_64 and aarch64 Linux, plus native
Darwin on macOS. Cargo selects the compiler, assembler, archiver, and
linker through target-specific environment variables.

Build GNU targets against a glibc 2.28 sysroot with static GCC runtimes
and use the musl toolchains for static Linux executables. Build Z3 and
AWS-LC with each target's stdenv and expose AWS-LC libraries and Rust
bindings through its target-specific system directory.

Keep Darwin system libraries on the unprocessed Apple SDK and prevent
the Rust toolchain from propagating replacement libraries into the shell.
Include the compiler and libc fixes needed for Darwin-to-Linux builds.

Share dependency and environment wiring through mkToolchain, keep compiler
wrappers in the Linux and Darwin modules, and group the pinned GNU build
environment under glibc-2.28. Document the toolchain boundaries in the build
architecture overview.

Validation: actionlint and nix fmt pass. The toolchain refactor preserves
the shell derivations for x86_64 Linux, aarch64 Linux, and aarch64 Darwin.

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
@SDAChess
SDAChess force-pushed the feat/nix-cross-compiling-and-sysroot branch from 6067054 to 2b2a48e Compare September 10, 2026 10:07
@SDAChess
SDAChess marked this pull request as ready for review September 10, 2026 10:08
Remove the dev-shell input and its matrix and workflow plumbing.
Use the host default shell for builds and cache hashing.

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
@github-actions

Copy link
Copy Markdown

Use target-specific artifact paths and rely on the Nix toolchains for
linkage. Remove post-link rewriting, platform linkage checks, and the
unused interpreter input. Update the build architecture documentation.

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
@SDAChess SDAChess added test:e2e Requires end-to-end coverage test:e2e-gpu Requires GPU end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage labels Sep 10, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e-gpu applied for a874eb8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute GPU E2E after building the required supervisor image once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@SDAChess

Copy link
Copy Markdown
Collaborator Author

/ok-to-test a874eb8

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for a874eb8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@github-actions

Copy link
Copy Markdown

Label test:e2e-kubernetes applied for a874eb8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

Use a pinned historical Nixpkgs recipe for glibc 2.28 and rebuild GCC 15
against it instead of maintaining separate runtime builds. Keep only
compatibility adjustments needed by the current build tools.

Assemble the sysroot from glibc outputs and static native libraries. Use
standard ELF interpreters and resolve Rust's explicit gcc_s dependency
through the static GCC archives.

Allow 90 minutes for Rust branch checks to accommodate cold toolchain
builds. Validate the Linux release matrix locally; Darwin remains for CI.

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Avoid building rpcgen against the Darwin SDK, which does not expose stat64.

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
@SDAChess
SDAChess enabled auto-merge September 10, 2026 14:26
@SDAChess
SDAChess added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit bcf96e4 Sep 10, 2026
93 of 98 checks passed
@SDAChess
SDAChess deleted the feat/nix-cross-compiling-and-sysroot branch September 10, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage test:e2e-gpu Requires GPU end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants